Weight Update Skipping: Reducing Training Time for Artificial Neural Networks
نویسندگان
چکیده
Artificial Neural Networks (ANNs) are known as state-of-the-art techniques in Machine Learning (ML) and have achieved outstanding results data-intensive applications, such recognition, classification, segmentation. These networks mostly use deep layers of convolution and/or fully connected with many filters each layer, demanding a large amount data tunable hyperparameters to achieve competitive accuracy. As result, storage, communication, computational costs training (in particular time spent for training) become limiting factors scale them up. In this paper, we propose new methodology ANNs that exploits the observation improvement accuracy shows temporal variations which allow us skip updating weights when variation is minuscule. During windows, keep bias ensures network still trains avoids overfitting; however, selectively (and their time-consuming computations). This approach virtually achieves same considerably less cost reduces on training. We developed two proposed method weights, call i) Weight Update Skipping (WUS), ii) Rate Scheduler (WUS+LR). evaluate these approaches by analyzing models, including AlexNet, VGG-11, VGG-16, ResNet-18 CIFAR datasets. also ImageNet dataset Resnet-18. On average, WUS WUS+LR reduced (compared baseline) 54%, 50% CPU 22%, 21% GPU, respectively CIFAR-10; 43% 35% CIFAR-100; finally 30% 27% ImageNet, respectively.
منابع مشابه
Reducing the Training Time of Neural Networks by Partitioning
This paper presents a new method for pre-training neural networks that can decrease the total training time for a neural network while maintaining the final performance, which motivates its use on deep neural networks. By partitioning the training task in multiple training subtasks with sub-models, which can be performed independently and in parallel, it is shown that the size of the sub-models...
متن کاملTraining Artificial Neural Networks for Fuzzy Logic
P roblems requiring inferencing with Boolean logic have been implemented in percept rons or feedforward networks, and some attempts have been made to implement fuzzy logic based inferencing in similar networks. In this pap er, we present producti ve networks , which are art ificial neur al networks, meant for fuzzy logic based inferencing. The nod es in t hese networks collect an offset product...
متن کاملAN EXTENDED FUZZY ARTIFICIAL NEURAL NETWORKS MODEL FOR TIME SERIES FORECASTING
Improving time series forecastingaccuracy is an important yet often difficult task.Both theoretical and empirical findings haveindicated that integration of several models is an effectiveway to improve predictive performance, especiallywhen the models in combination are quite different. In this paper,a model of the hybrid artificial neural networks andfuzzy model is proposed for time series for...
متن کاملArtificial Bee Colony Training of Neural Networks
The Artificial Bee Colony (ABC) is a recently introduced swarm intelligence algorithm for optimization, that has previously been applied successfully to the training of neural networks. This paper explores more carefully the performance of the ABC algorithm for optimizing the connection weights of feed-forward neural networks for classification tasks, and presents a more rigorous comparison wit...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: IEEE Journal on Emerging and Selected Topics in Circuits and Systems
سال: 2021
ISSN: ['2156-3365', '2156-3357']
DOI: https://doi.org/10.1109/jetcas.2021.3127907